Predict probability whether a given ticket will be paid?¶

Blight violations in the city of Detroit has been an dificult problem to solve. Every year, the city of Detroit issues millions of dollars in fines to residents and every year, many of these fines remain unpaid. Enforcing unpaid blight fines is a costly and tedious process, so the city wants to know: how can we increase blight ticket compliance?

Blighted property shall include commercial or residential structures, which have been declared uninhabitable and hazardous by the City and exhibit objectively determinable signs of deterioration that constitute a threat to human health, safety and public welfare.

In this notebook, I will focus on the problem at hand, helping the city enforce fines by predicting whether a person is going to comply with a property maintenance fine or not.

The first step in answering this question is understanding when and why a resident might fail to comply with a blight ticket. This is where predictive modeling comes in. For this assignment, your task is to predict whether a given blight ticket will be paid on time.

The Dataset¶

I have two data files for use in training and validating my models: train.csv and test.csv. Each row in these two files corresponds to a single blight ticket, and includes information about when, why, and to whom each ticket was issued.

The target variable is compliance, which is True if the ticket was paid early, on time, or within one month of the hearing data, False if the ticket was paid after the hearing date or not at all, and Null if the violator was found not responsible.

Note: All tickets where the violators were found not responsible are not considered during evaluation. They are included in the training set as an additional source of data for visualization, and to enable unsupervised and semi-supervised approaches. However, they are not included in the test set.


File descriptions (used for training model!)

train.csv - the training set (all tickets issued 2004-2011)
test.csv - the test set (all tickets issued 2012-2016)
addresses.csv & latlons.csv - mapping from ticket id to addresses, and from addresses to lat/lon coordinates. 
 Note: misspelled addresses may be incorrectly geolocated.


Data fields

train.csv & test.csv

ticket_id - unique identifier for tickets
agency_name - Agency that issued the ticket
inspector_name - Name of inspector that issued the ticket
violator_name - Name of the person/organization that the ticket was issued to
violation_street_number, violation_street_name, violation_zip_code - Address where the violation occurred
mailing_address_str_number, mailing_address_str_name, city, state, zip_code, non_us_str_code, country - Mailing address of the violator
ticket_issued_date - Date and time the ticket was issued
hearing_date - Date and time the violator's hearing was scheduled
violation_code, violation_description - Type of violation
disposition - Judgment and judgement type
fine_amount - Violation fine amount, excluding fees
admin_fee - $20 fee assigned to responsible judgments

state_fee - $10 fee assigned to responsible judgments late_fee - 10% fee assigned to responsible judgments discount_amount - discount applied, if any clean_up_cost - DPW clean-up or graffiti removal cost judgment_amount - Sum of all fines and fees grafitti_status - Flag for graffiti violations

train.csv only

payment_amount - Amount paid, if any
payment_date - Date payment was made, if it was received
payment_status - Current payment status as of Feb 1 2017
balance_due - Fines and fees still owed
collection_status - Flag for payments in collections
compliance [target variable for prediction] 
 Null = Not responsible
 0 = Responsible, non-compliant
 1 = Responsible, compliant
compliance_detail - More information on why each ticket was marked compliant or non-compliant

Evaluation¶

The predictions are given as the probability that the corresponding blight ticket will be paid on time.

The evaluation metric for this assignment is the Area Under the ROC Curve (AUC).

Import Libraries¶

In [1]:
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings("ignore")
# Importing re package
import re

import seaborn as sns
import matplotlib.pyplot as plt
In [2]:
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import GridSearchCV
In [3]:
%matplotlib inline

class color:
   BLUE = '\033[94m'
   PURPLE = '\033[95m'
   RED = '\033[91m'
   BOLD = '\033[1m'
   END = '\033[0m'

Load Test Train Data¶

In [4]:
train_data = pd.read_csv('train.csv', encoding = 'ISO-8859-1')
test_data = pd.read_csv('test.csv', encoding = 'ISO-8859-1')

Data Overview ¶

Let us look at the data.

In [5]:
def overview(df):
    print(color.BOLD + "There are {} rows and {} columns in the dataset.".format(df.shape[0],df.shape[1]),"\n"+ color.END)
    print(color.BOLD +color.BLUE +"Let's look at the data types available in the dataset"+ color.END)
    df.info()
In [6]:
overview(train_data)
print(color.BOLD +color.BLUE +"\n","Summary statistics of dataset"+ color.END)
train_data.describe()

There are 250306 rows and 34 columns in the dataset. 

Let's look at the data types available in the dataset
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 250306 entries, 0 to 250305
Data columns (total 34 columns):
 #   Column                      Non-Null Count   Dtype  
---  ------                      --------------   -----  
 0   ticket_id                   250306 non-null  int64  
 1   agency_name                 250306 non-null  object 
 2   inspector_name              250306 non-null  object 
 3   violator_name               250272 non-null  object 
 4   violation_street_number     250306 non-null  float64
 5   violation_street_name       250306 non-null  object 
 6   violation_zip_code          0 non-null       float64
 7   mailing_address_str_number  246704 non-null  float64
 8   mailing_address_str_name    250302 non-null  object 
 9   city                        250306 non-null  object 
 10  state                       250213 non-null  object 
 11  zip_code                    250305 non-null  object 
 12  non_us_str_code             3 non-null       object 
 13  country                     250306 non-null  object 
 14  ticket_issued_date          250306 non-null  object 
 15  hearing_date                237815 non-null  object 
 16  violation_code              250306 non-null  object 
 17  violation_description       250306 non-null  object 
 18  disposition                 250306 non-null  object 
 19  fine_amount                 250305 non-null  float64
 20  admin_fee                   250306 non-null  float64
 21  state_fee                   250306 non-null  float64
 22  late_fee                    250306 non-null  float64
 23  discount_amount             250306 non-null  float64
 24  clean_up_cost               250306 non-null  float64
 25  judgment_amount             250306 non-null  float64
 26  payment_amount              250306 non-null  float64
 27  balance_due                 250306 non-null  float64
 28  payment_date                41113 non-null   object 
 29  payment_status              250306 non-null  object 
 30  collection_status           36897 non-null   object 
 31  grafitti_status             1 non-null       object 
 32  compliance_detail           250306 non-null  object 
 33  compliance                  159880 non-null  float64
dtypes: float64(13), int64(1), object(20)
memory usage: 64.9+ MB

 Summary statistics of dataset
Out[6]:
ticket_id violation_street_number violation_zip_code mailing_address_str_number fine_amount admin_fee state_fee late_fee discount_amount clean_up_cost judgment_amount payment_amount balance_due compliance
count 250306.000000 2.503060e+05 0.0 2.467040e+05 250305.000000 250306.000000 250306.000000 250306.000000 250306.000000 250306.0 250306.000000 250306.000000 250306.000000 159880.000000
mean 152665.543099 1.064986e+04 NaN 9.149788e+03 374.423435 12.774764 6.387382 21.494506 0.125167 0.0 268.685356 48.898986 222.449058 0.072536
std 77189.882881 3.188733e+04 NaN 3.602034e+04 707.195807 9.607344 4.803672 56.464263 3.430178 0.0 626.915212 222.422425 606.394010 0.259374
min 18645.000000 0.000000e+00 NaN 1.000000e+00 0.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 -7750.000000 0.000000
25% 86549.250000 4.739000e+03 NaN 5.440000e+02 200.000000 0.000000 0.000000 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 0.000000
50% 152597.500000 1.024400e+04 NaN 2.456000e+03 250.000000 20.000000 10.000000 10.000000 0.000000 0.0 140.000000 0.000000 25.000000 0.000000
75% 219888.750000 1.576000e+04 NaN 1.292725e+04 250.000000 20.000000 10.000000 25.000000 0.000000 0.0 305.000000 0.000000 305.000000 0.000000
max 366178.000000 1.415411e+07 NaN 5.111345e+06 10000.000000 20.000000 10.000000 1000.000000 350.000000 0.0 11030.000000 11075.000000 11030.000000 1.000000
In [7]:
overview(test_data)
print(color.BOLD +color.BLUE +"\n","Summary statistics of dataset"+ color.END)
test_data.describe()

There are 61001 rows and 27 columns in the dataset. 

Let's look at the data types available in the dataset
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 61001 entries, 0 to 61000
Data columns (total 27 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   ticket_id                   61001 non-null  int64  
 1   agency_name                 61001 non-null  object 
 2   inspector_name              61001 non-null  object 
 3   violator_name               60973 non-null  object 
 4   violation_street_number     61001 non-null  float64
 5   violation_street_name       61001 non-null  object 
 6   violation_zip_code          24024 non-null  object 
 7   mailing_address_str_number  59987 non-null  object 
 8   mailing_address_str_name    60998 non-null  object 
 9   city                        61000 non-null  object 
 10  state                       60670 non-null  object 
 11  zip_code                    60998 non-null  object 
 12  non_us_str_code             0 non-null      float64
 13  country                     61001 non-null  object 
 14  ticket_issued_date          61001 non-null  object 
 15  hearing_date                58804 non-null  object 
 16  violation_code              61001 non-null  object 
 17  violation_description       61001 non-null  object 
 18  disposition                 61001 non-null  object 
 19  fine_amount                 61001 non-null  float64
 20  admin_fee                   61001 non-null  float64
 21  state_fee                   61001 non-null  float64
 22  late_fee                    61001 non-null  float64
 23  discount_amount             61001 non-null  float64
 24  clean_up_cost               61001 non-null  float64
 25  judgment_amount             61001 non-null  float64
 26  grafitti_status             2221 non-null   object 
dtypes: float64(9), int64(1), object(17)
memory usage: 12.6+ MB

 Summary statistics of dataset
Out[7]:
ticket_id violation_street_number non_us_str_code fine_amount admin_fee state_fee late_fee discount_amount clean_up_cost judgment_amount
count 61001.000000 6.100100e+04 0.0 61001.000000 61001.0 61001.0 61001.000000 61001.000000 61001.000000 61001.000000
mean 331724.532811 1.256638e+04 NaN 272.714185 20.0 10.0 25.116219 0.239340 20.649711 347.895541
std 25434.932141 1.414373e+05 NaN 360.101855 0.0 0.0 36.310155 3.245894 242.375180 460.058043
min 284932.000000 -1.512600e+04 NaN 0.000000 20.0 10.0 0.000000 0.000000 0.000000 0.000000
25% 310111.000000 6.008000e+03 NaN 50.000000 20.0 10.0 5.000000 0.000000 0.000000 85.000000
50% 332251.000000 1.213400e+04 NaN 200.000000 20.0 10.0 10.000000 0.000000 0.000000 250.000000
75% 353031.000000 1.716500e+04 NaN 250.000000 20.0 10.0 25.000000 0.000000 0.000000 305.000000
max 376698.000000 2.010611e+07 NaN 10000.000000 20.0 10.0 1000.000000 250.000000 15309.000000 15558.800000

Load Address Data and Create a map with property location¶

In [8]:
from geopy.geocoders import Nominatim
from folium.plugins import HeatMap
import folium
from tqdm import tqdm
In [9]:
#load datasets
addresses =  pd.read_csv('addresses.csv')
lat_lons = pd.read_csv('latlons.csv')
In [10]:
#join datasets
id_address= addresses.set_index('address').join(lat_lons.set_index('address')).reset_index().set_index('ticket_id')
In [11]:
print(color.BOLD + "There are {} rows and {} columns in the dataset.".format(id_address.shape[0],id_address.shape[1]),"\n"+ color.END)

There are 311307 rows and 3 columns in the dataset.
In [12]:
print(color.BOLD +color.BLUE +"\n","Summary statistics of dataset"+ color.END)
id_address.describe()


 Summary statistics of dataset
Out[12]:
lat lon
count 311299.000000 311299.000000
mean 42.389822 -83.112668
std 0.038155 0.095320
min 41.785926 -88.081348
25% 42.364200 -83.188845
50% 42.393794 -83.126422
75% 42.419709 -83.036160
max 45.809387 -82.433593
In [13]:
address_locations=pd.DataFrame(id_address['address'].value_counts().reset_index())
address_locations.columns=['address','count']
In [14]:
address_locations=address_locations.merge(id_address,on='address',how="left").dropna()
address_locations['count'].max()
Out[14]:
203
In [15]:
def defaultmap(default_location=[42.3314, -83.0458], default_zoom_start=9):
    base_map = folium.Map(location=default_location, control_scale=True, zoom_start=default_zoom_start)
    return base_map
In [16]:
basemap=defaultmap()
HeatMap(address_locations[['lat','lon','count']].values.tolist(),zoom=20,radius=15).add_to(basemap)
Out[16]:
<folium.plugins.heat_map.HeatMap at 0x286dbb2e8f0>
In [17]:
basemap
Out[17]:
Make this Notebook Trusted to load map: File -> Trust Notebook

Remove train only columns from train dataset ¶

In [18]:
train_only_columns = ['payment_amount', 'payment_date', 'payment_status', 'balance_due','collection_status', 'compliance_detail']
train_data.drop(train_only_columns, axis=1, inplace=True)

get the unique values form variaous categorical variables from both test and train datasets

</span>

In [19]:
def uniqueplot(df):
    data_ = df.select_dtypes(exclude=['int', 'float'])
    df_variable_unique_levels_=(pd.DataFrame(data_.nunique())).reset_index().rename(columns={'index':'Variable', 0:'Unique_Levels'})
    df_variable_unique_levels_.sort_values('Unique_Levels',ascending=False, inplace=True)

    # Create just a figure and only one subplot
    fig, ax = plt.subplots(figsize=(10,10))

    ax = sns.barplot(x="Variable", y="Unique_Levels", data=df_variable_unique_levels_)

    # rotates labels and aligns them horizontally to left 
    plt.setp( ax.xaxis.get_majorticklabels(), rotation=90, ha="left" )

    plt.tight_layout()
    plt.show()
In [20]:
### Train Data
uniqueplot(train_data)
In [21]:
### Test Data
uniqueplot(test_data)

variables such as inspector_name, violator_name, violation_street_name, mailing_address_str_name, city, state, zip_code, country, violation_code, violation_description, have too many unique values.
These might not provide any useful information, so we can delete these columns. But before that we can use the necessary information from these columns to convert them into required and delete them.

Remove variables with too many unique values ¶

Drop inspector_name, violator_name, violation_street_name, mailing_address_str_name, zip_code, violation_description from test and train

In [22]:
# Remove column name from both test and train
cols_del=['inspector_name', 'violator_name', 'violation_street_name', 'mailing_address_str_name', 'zip_code', 'violation_description']
train_data.drop(cols_del, axis=1, inplace=True)
test_data.drop(cols_del, axis=1, inplace=True)

Feature Engineering ¶

Convert Categorical variables to Binary Variables¶

In [23]:
def converttobinary(df):
    print(color.BOLD +color.BLUE +"\n","country Column","\n"+ color.END)
    print(color.BOLD +color.PURPLE +"If the violator statying is within country USA or not, If the violator is statying within USA 1 else 0"+ color.END)
    print(color.BLUE +"Unique values in country column"+ color.END)
    country_df_valcounts_=pd.DataFrame(df["country"].value_counts()).rename(columns={'country':'Total Rows'})
    print(country_df_valcounts_)
    print(color.RED +"There are {} unique values in country column".format(country_df_valcounts_.count()[0])+ color.END)
    df = df[df.country == 'USA']
    df['USA'] = df['country'].apply(lambda x: 1 if 'USA' else 0)
    
    print(color.BOLD +color.BLUE +"\n","city Column","\n"+ color.END)
    print(color.BOLD +color.PURPLE +"If the violator is statying within city detriot, MI or not, If the violator is statying within city detriot 1 else 0"+ color.END)
    df['within_Detriot'] = np.where((df.state == 'MI' ) & (df['city'].str.findall('det', flags=re.IGNORECASE)), 1, 0)
    
    print(color.BOLD +color.BLUE +"\n","state Column","\n"+ color.END)
    print(color.BOLD +color.PURPLE +"If the violator is statying within state, MI or not, if the violator is statying within state 1 else 0"+ color.END)
    df['MI'] = np.where((df.state == 'MI' ), 1, 0)
In [24]:
## Train data
converttobinary(train_data)


 country Column 

If the violator statying is within country USA or not, If the violator is statying within USA 1 else 0
Unique values in country column
      Total Rows
USA       250293
Cana           7
Aust           3
Egyp           2
Germ           1
There are 5 unique values in country column

 city Column 

If the violator is statying within city detriot, MI or not, If the violator is statying within city detriot 1 else 0

 state Column 

If the violator is statying within state, MI or not, if the violator is statying within state 1 else 0
In [25]:
## Test data
converttobinary(test_data)
print("\n")
print("Drop country, city,state column from test and train data")


 country Column 

If the violator statying is within country USA or not, If the violator is statying within USA 1 else 0
Unique values in country column
     Total Rows
USA       61001
There are 1 unique values in country column

 city Column 

If the violator is statying within city detriot, MI or not, If the violator is statying within city detriot 1 else 0

 state Column 

If the violator is statying within state, MI or not, if the violator is statying within state 1 else 0


Drop country, city,state column from test and train data
In [26]:
### get the top 10 violation codes and if the violation is in top 10 then 1 or else 0 for both test and train data.
print(color.BOLD +color.BLUE +"\n","violation codes Column","\n"+ color.END)
    
# train data 
top10_violation_types = train_data.violation_code.value_counts().index[0:10]
train_data['top10_violation_types'] = [1 if c in top10_violation_types else 0 for c in train_data.violation_code]

# test data 
test_data['top10_violation_types'] = [1 if c in top10_violation_types else 0 for c in test_data.violation_code]


 violation codes Column

Drop unwanted columns ¶

Drop country, city, state column from test and train data¶

In [27]:
delete_columns=['country', 'city', 'state','violation_code']
train_data.drop(delete_columns, axis=1, inplace=True)
test_data.drop(delete_columns, axis=1, inplace=True)

Drop columns nan_columns ¶

In [28]:
#Pandas will recognize both empty cells and “NA” types as missing values. 
# there are many empty rows in grafitti_status we can delete the column
nan_columns=[ 'violation_street_number', 'violation_zip_code', 'mailing_address_str_number', 'non_us_str_code', 'violation_zip_code', 'non_us_str_code', 'grafitti_status']
train_data.drop(nan_columns, axis=1, inplace=True)
test_data.drop(nan_columns, axis=1, inplace=True)

More EDA ¶

agency_name¶

In [29]:
agency_train_valcounts_=pd.DataFrame(train_data["agency_name"].value_counts()).rename(columns={'agency_name':'Total Rows'})
agency_train_valcounts_
Out[29]:
Total Rows
Buildings, Safety Engineering & Env Department 157784
Department of Public Works 74717
Health Department 8903
Detroit Police Department 8900
Neighborhood City Halls 2
In [30]:
agency_test_valcounts_=pd.DataFrame(test_data["agency_name"].value_counts()).rename(columns={'agency_name':'Total Rows'})
agency_test_valcounts_
Out[30]:
Total Rows
Department of Public Works 40731
Buildings, Safety Engineering & Env Department 16832
Detroit Police Department 3438
In [31]:
agency_compliance_=pd.crosstab(train_data.agency_name, train_data.compliance)
In [32]:
agency_compliance_
Out[32]:
compliance 0.0 1.0
agency_name
Buildings, Safety Engineering & Env Department 90040 5823
Department of Public Works 47727 4718
Detroit Police Department 3876 588
Health Department 6639 468
Neighborhood City Halls 1 0
In [33]:
pd.merge(agency_compliance_, agency_train_valcounts_, left_index=True, right_index=True).reset_index().rename(columns={'index':'Agency_Name', 0.0:'Non_Compliant_0', 1.0:'Compliant_1'})
Out[33]:
Agency_Name Non_Compliant_0 Compliant_1 Total Rows
0 Buildings, Safety Engineering & Env Department 90040 5823 157784
1 Department of Public Works 47727 4718 74717
2 Detroit Police Department 3876 588 8900
3 Health Department 6639 468 8903
4 Neighborhood City Halls 1 0 2
In [34]:
# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(10,10))

color= ['dodgerblue', 'hotpink']
sns.countplot(hue ='compliance', x = "agency_name", data = train_data, palette=color)

# rotates labels and aligns them horizontally to left 
plt.setp( ax.xaxis.get_majorticklabels(), rotation=-45, ha="left" )

plt.tight_layout()
plt.show()

Through there are only three unique agency_names test data, from the train data it can be observed that by deleting the agency_names that are not in test data we might loose some information regarding the target variable. Particularly for Health Department, but there is one row in Neighborhood City Halls which is not compliant, we can delete those corresponding to Neighborhood City Hall from test data

In [35]:
delete_list=['Neighborhood City Halls']

train_data = train_data[~train_data['agency_name'].isin(delete_list)]

disposition¶

In [36]:
disposition_train_valcounts_=pd.DataFrame(train_data["disposition"].value_counts()).rename(columns={'disposition':'Total Rows'})
disposition_train_valcounts_
Out[36]:
Total Rows
Responsible by Default 138340
Not responsible by Dismissal 48694
Not responsible by City Dismissal 34401
Responsible by Admission 13701
Responsible by Determination 7643
Not responsible by Determination 6639
PENDING JUDGMENT 387
SET-ASIDE (PENDING JUDGMENT) 304
Responsible (Fine Waived) by Deter 195
In [37]:
disposition_test_valcounts_=pd.DataFrame(test_data["disposition"].value_counts()).rename(columns={'disposition':'Total Rows'})
disposition_test_valcounts_
Out[37]:
Total Rows
Responsible by Default 51602
Responsible by Admission 4484
Responsible by Determination 4124
Responsible (Fine Waived) by Deter 781
Responsible - Compl/Adj by Default 6
Responsible - Compl/Adj by Determi 2
Responsible (Fine Waived) by Admis 1
Responsible by Dismissal 1

In the train data there are some more columns where the disposition is under not responsible, we can delete the

In [38]:
disposition_compliance_=pd.crosstab(train_data.disposition, train_data.compliance)
disposition_compliance_
Out[38]:
compliance 0.0 1.0
disposition
Responsible (Fine Waived) by Deter 0 195
Responsible by Admission 9933 3768
Responsible by Default 133055 5285
Responsible by Determination 5294 2349

looking at the disposition and compliance cross tab we can usderstand that Responsible by Determination, Responsible by Default, Responsible by Admission, Responsible (Fine Waived) by Deter are most important. We can delete those where it is not responsible.

In [39]:
delete_list=['Not responsible by Dismissal', 'Not responsible by City Dismissal', 'Not responsible by Determination',
             'PENDING JUDGMENT','SET-ASIDE (PENDING JUDGMENT)']

train_data = train_data[~train_data['disposition'].isin(delete_list)]

after deletion do one hot encoding for remaining under disposition and drop the columns Responsible - Compl/Adj by Default,Responsible - Compl/Adj by Determi,Responsible (Fine Waived) by Admis,Responsible by Dismissal from the test data

Relation between numeric variables ¶

  • state_fee and admin_fee is same irrespective of compliance hence-ignore those columns

  • clean_up_cost is mostly 0-ignore the column

  • discount_amount there are very few rows with discount amount-ignore that column

  • judgment_amount is sum of fine amount and other fee-ignore that column

In [40]:
sns.scatterplot(data=train_data, x="fine_amount", y="late_fee")

plt.show() 

corr_val = train_data[['fine_amount','late_fee']].corr()
corr_val
Out[40]:
fine_amount late_fee
fine_amount 1.000000 0.986787
late_fee 0.986787 1.000000

there is high correlation between the fine_amount and late fee, based on the correlation either of fine_amount and late_fee will be used for the model building

fine_amount¶

In [41]:
# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(10,10))

sns.boxplot(x='compliance',y='fine_amount',data=train_data, palette='rainbow')

plt.tight_layout()
plt.show()
In [42]:
#Perform mean and median fine_amount
fine_amount_agg = train_data.groupby('compliance').fine_amount.agg(['mean','median','std','var']).reset_index()
fine_amount_agg
Out[42]:
compliance mean median std var
0 0.0 366.315008 250.0 693.570453 481039.972606
1 1.0 238.327240 250.0 357.343654 127694.486989

there is a significant difference between the means of the fine amount between compliant and non compliant

Late fee¶

In [43]:
# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(10,10))

colors=['blue','lightcoral']
sns.boxplot(x='compliance',y='late_fee',data=train_data, palette=colors)

plt.tight_layout()
plt.show()
In [44]:
#Perform mean and median late_fee
late_fee_agg = train_data.groupby('compliance').late_fee.agg(['mean','median','std','var']).reset_index()
late_fee_agg
Out[44]:
compliance mean median std var
0 0.0 35.26124 25.0 69.498911 4830.098653
1 1.0 13.06343 5.0 30.942405 957.432434

there is a significant difference between the means of the late fee between compliant and non compliant

Detecting Multicollinearity with VIF¶

Multicollinearity occurs when there are two or more independent variables in a multi regression model, which have a high correlation among themselves. When some features are highly correlated, we might have difficulty in distinguishing between their individual effects on the target variable.

In VIF method, we pick each feature and regress it against all of the other features. For each regression, the factor is calculated as :

VIF=\frac{1}{1-R^2}

Where, R-squared is the coefficient of determination in linear regression. Its value lies between 0 and 1.

As we see from the formula, greater the value of R-squared, greater is the VIF. Hence, greater VIF denotes greater correlation. This is in agreement with the fact that a higher R-squared value denotes a stronger collinearity. Generally, a VIF above 5 indicates a high multicollinearity.

Dont consider for VIF¶

  • dependent variable('compliance'), and columns ticket_id, ticket_issued_date, hearing_date

  • clean_up_cost as there are more nan in the column

  • agency_name, disposition as they are categorical variables

In [45]:
training_vif= train_data.drop(['compliance', 'ticket_id', 'ticket_issued_date', 'hearing_date', 'clean_up_cost', 'agency_name', 'disposition'], axis='columns')

# VIF dataframe

from statsmodels.stats.outliers_influence import variance_inflation_factor

vif_data = pd.DataFrame()
vif_data["feature"] = training_vif.columns
  
# calculating VIF for each feature
vif_data["VIF"] = [variance_inflation_factor(training_vif.values, i)
                   for i in range(len(training_vif.columns))]
  
print(vif_data)

                 feature            VIF
0            fine_amount  416652.246629
1              admin_fee       0.000000
2              state_fee       0.000000
3               late_fee    4221.034608
4        discount_amount       1.171671
5        judgment_amount  464586.439750
6  top10_violation_types       1.000398

  • as mentioined judgment_amount is sum of fine_amount and other fee, we can expect multicollinearity with other variables. This is reflecting thtrough vif study

  • judgment_amount has a very high vif, hence we calculate the vif for other variables again after removing the vaiable

  • after removing judgment_amount vif for other variables has reduced significantly

  • as seen earlier, there is high correlation between fie_amount and late_fee

  • in the second iteration we remove the late_fee and check the vif again for rest of the variables.

  • Drop judgement_amount and fine_amount

In [46]:
training_vif_itr= training_vif.drop(['judgment_amount', 'late_fee'], axis='columns')

vif_data_itr = pd.DataFrame()
vif_data_itr["feature"] = training_vif_itr.columns
  
# calculating VIF for each feature
vif_data_itr["VIF"] = [variance_inflation_factor(training_vif_itr.values, i)
                     for i in range(len(training_vif_itr.columns))]
  
print(vif_data_itr)
# vif after dropping two variables judgement_fee and late_fee
# this is final patch after checking the vif individually.

                 feature       VIF
0            fine_amount  1.001959
1              admin_fee  0.000000
2              state_fee  0.000000
3        discount_amount  1.001603
4  top10_violation_types  1.000385

checking the dfs' is there are any more null¶

In [47]:
# printing column name where null is present
print()
print("printing column name where null is present")
col_name = train_data.isnull().any()
print(col_name)

printing column name where null is present
ticket_id                False
agency_name              False
ticket_issued_date       False
hearing_date              True
disposition              False
fine_amount              False
admin_fee                False
state_fee                False
late_fee                 False
discount_amount          False
clean_up_cost            False
judgment_amount          False
compliance               False
top10_violation_types    False
dtype: bool
In [48]:
# printing column name where null is present
print()
print("printing column name where null is present")
col_name = test_data.isnull().any()
print(col_name)

printing column name where null is present
ticket_id                False
agency_name              False
ticket_issued_date       False
hearing_date              True
disposition              False
fine_amount              False
admin_fee                False
state_fee                False
late_fee                 False
discount_amount          False
clean_up_cost            False
judgment_amount          False
top10_violation_types    False
dtype: bool

convert the dates to usable manner¶

In [49]:
from datetime import date
train_data[['ticket_issued_date','hearing_date']] = train_data[['ticket_issued_date','hearing_date']].apply(pd.to_datetime)
test_data[['ticket_issued_date','hearing_date']] = test_data[['ticket_issued_date','hearing_date']].apply(pd.to_datetime) 

# forward fill to fill the gaps in hearing date
train_data.hearing_date.fillna(method='ffill', inplace=True)
test_data.hearing_date.fillna(method='ffill', inplace=True)
In [50]:
# we take the difference between the ticket_issued_date and hearing_date in number of days
train_data['ticket_hearing_gap'] = (train_data['hearing_date'] - train_data['ticket_issued_date']).dt.days
In [51]:
# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(10,10))

colors=['deeppink','aqua']
sns.boxplot(x='compliance',y='ticket_hearing_gap',data=train_data, palette=colors)

plt.tight_layout()
plt.show()
In [52]:
#Perform mean and median ticket_hearing_gap
ticket_hearing_gap_agg = train_data.groupby('compliance').ticket_hearing_gap.agg(['mean','median','std','var']).reset_index()
ticket_hearing_gap_agg
Out[52]:
compliance mean median std var
0 0.0 72.724727 54.0 59.246169 3510.108550
1 1.0 71.544020 48.0 96.127696 9240.533872
In [53]:
test_data['ticket_hearing_gap'] = (test_data['hearing_date'] - test_data['ticket_issued_date']).dt.days

Drop ticket_issued_date and hearing_date

Drop unwanted columns¶

Drop ticket_id,clean_up_cost ,judgment_amount, late_fee, ticket_issued_date and hearing_date¶

In [54]:
delete_columns=['clean_up_cost' ,'judgment_amount', 'late_fee', 'ticket_issued_date', 'hearing_date']
train_data.drop(delete_columns, axis=1, inplace=True)
test_data.drop(delete_columns, axis=1, inplace=True)
In [55]:
train_data.reset_index(drop=True, inplace=True)
test_data.reset_index(drop=True, inplace=True)

Dummy Variable Creation¶

In [56]:
#Checking for the labels in the categorical parameters
print('unique values in agency_name')
print(train_data['agency_name'].value_counts())

unique values in agency_name
Buildings, Safety Engineering & Env Department    95863
Department of Public Works                        52445
Health Department                                  7107
Detroit Police Department                          4464
Name: agency_name, dtype: int64
In [57]:
print('unique values in disposition')
print(train_data['disposition'].value_counts())

unique values in disposition
Responsible by Default                138340
Responsible by Admission               13701
Responsible by Determination            7643
Responsible (Fine Waived) by Deter       195
Name: disposition, dtype: int64
In [58]:
# One-Hot encoding the categorical parameters using get_dummies()
training_df = pd.get_dummies(train_data, columns = ['agency_name', 'disposition'])
training_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 159879 entries, 0 to 159878
Data columns (total 16 columns):
 #   Column                                                      Non-Null Count   Dtype  
---  ------                                                      --------------   -----  
 0   ticket_id                                                   159879 non-null  int64  
 1   fine_amount                                                 159879 non-null  float64
 2   admin_fee                                                   159879 non-null  float64
 3   state_fee                                                   159879 non-null  float64
 4   discount_amount                                             159879 non-null  float64
 5   compliance                                                  159879 non-null  float64
 6   top10_violation_types                                       159879 non-null  int64  
 7   ticket_hearing_gap                                          159879 non-null  int64  
 8   agency_name_Buildings, Safety Engineering & Env Department  159879 non-null  uint8  
 9   agency_name_Department of Public Works                      159879 non-null  uint8  
 10  agency_name_Detroit Police Department                       159879 non-null  uint8  
 11  agency_name_Health Department                               159879 non-null  uint8  
 12  disposition_Responsible (Fine Waived) by Deter              159879 non-null  uint8  
 13  disposition_Responsible by Admission                        159879 non-null  uint8  
 14  disposition_Responsible by Default                          159879 non-null  uint8  
 15  disposition_Responsible by Determination                    159879 non-null  uint8  
dtypes: float64(5), int64(3), uint8(8)
memory usage: 11.0 MB
In [59]:
# One-Hot encoding the categorical parameters using get_dummies()
testing_df = pd.get_dummies(test_data, columns = ['agency_name', 'disposition'])
testing_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 61001 entries, 0 to 61000
Data columns (total 18 columns):
 #   Column                                                      Non-Null Count  Dtype  
---  ------                                                      --------------  -----  
 0   ticket_id                                                   61001 non-null  int64  
 1   fine_amount                                                 61001 non-null  float64
 2   admin_fee                                                   61001 non-null  float64
 3   state_fee                                                   61001 non-null  float64
 4   discount_amount                                             61001 non-null  float64
 5   top10_violation_types                                       61001 non-null  int64  
 6   ticket_hearing_gap                                          61001 non-null  int64  
 7   agency_name_Buildings, Safety Engineering & Env Department  61001 non-null  uint8  
 8   agency_name_Department of Public Works                      61001 non-null  uint8  
 9   agency_name_Detroit Police Department                       61001 non-null  uint8  
 10  disposition_Responsible (Fine Waived) by Admis              61001 non-null  uint8  
 11  disposition_Responsible (Fine Waived) by Deter              61001 non-null  uint8  
 12  disposition_Responsible - Compl/Adj by Default              61001 non-null  uint8  
 13  disposition_Responsible - Compl/Adj by Determi              61001 non-null  uint8  
 14  disposition_Responsible by Admission                        61001 non-null  uint8  
 15  disposition_Responsible by Default                          61001 non-null  uint8  
 16  disposition_Responsible by Determination                    61001 non-null  uint8  
 17  disposition_Responsible by Dismissal                        61001 non-null  uint8  
dtypes: float64(4), int64(3), uint8(11)
memory usage: 3.9 MB
In [60]:
train_ticketid=training_df[['ticket_id']]
test_ticketid=testing_df[['ticket_id']]

drop agency_name_Health Department which is not in testing df.

In [61]:
training_df.drop(['ticket_id', 'agency_name_Health Department'], axis=1, inplace=True)

drop the columns Responsible - Compl/Adj by Default,Responsible - Compl/Adj by Determi,Responsible (Fine Waived) by Admis,Responsible by Dismissal from the test data as the relevant information is not available in the train data

In [62]:
delete_columns=['ticket_id',
                'disposition_Responsible (Fine Waived) by Admis',
                'disposition_Responsible - Compl/Adj by Default',
                'disposition_Responsible - Compl/Adj by Determi',
                'disposition_Responsible by Dismissal']

testing_df.drop(delete_columns, axis=1, inplace=True)
In [63]:
### rename columns

col_name={'agency_name_Buildings, Safety Engineering & Env Department': 'Agency_BSE',
          'agency_name_Department of Public Works': 'Agency_DPW',
          'agency_name_Detroit Police Department': 'Agency_DPD',
          'disposition_Responsible (Fine Waived) by Deter': 'Dis_res_deter',
          'disposition_Responsible by Admission':'Dis_res_admission',
          'disposition_Responsible by Default':'Dis_res_default',
          'disposition_Responsible by Determination':'Dis_res_determination'}

training_df.rename(columns=col_name, inplace=True)
testing_df.rename(columns=col_name, inplace=True)

Handling Class imbalance¶

In [64]:
compliance_=pd.DataFrame(training_df["compliance"].value_counts().apply(lambda x: x/len(training_df["compliance"]))).rename(columns={'compliance':'percentage of class'}).T
compliance_
Out[64]:
0.0 1.0
percentage of class 0.927464 0.072536
In [65]:
# Create just a figure and only one subplot
fig, ax = plt.subplots(figsize=(5,4))

colors=['deeppink','dodgerblue']
sns.countplot(x='compliance',data=train_data, palette=colors)

plt.title("Distribution of target class")
plt.ylabel("count")

plt.tight_layout()
plt.show()

There is a huge difference between the class of target variables. 148273 non-compliant transactions and 11595 compliant. When there is a huge imbalance in target class, due to which classifier will always predict majority class. Though the accuracy is very high in such scenario, it is misleading.

We can use the dummy classifier to understand the performance of the actual data set with class imbalance

Dummy classifier that classifies everything as the majority class of the training data¶

In [66]:
# Simultaneously we will add the scores to a df for future reference

scores = pd.DataFrame(columns = ['Accuracy', 'Recall score', 'Precision score', 'F1 score', 'Specificity', 'AUC','TPR','FPR'])
In [67]:
#create a function to add rows
def add_row(scores, row):
    scores.loc[-1] = row
    scores.index = scores.index + 1
    return scores
In [68]:
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, recall_score, precision_score, roc_curve, auc, f1_score,roc_auc_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
In [69]:
# default is 75% / 25% train-test split
X = training_df.drop('compliance', axis=1)
y = training_df.compliance

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
In [70]:
# use most frequent strategy always predicts '0'
dclf = DummyClassifier(strategy = 'most_frequent', random_state = 0)

dclf.fit(X_train, y_train)

#get predictions 
y_pred = dclf.predict(X_test)

# predict probabilities
y_probs = dclf.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)
#roc_auc_ = auc(fpr_, tpr_)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)

Though the accuracy of the dummy classifier is very high, based on the recall and precision score we can understand that true positives is 0.

In [71]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Synthetic Minority Oversampling Technique (SMOTE)¶

This technique generates synthetic data for the minority class.

SMOTE (Synthetic Minority Oversampling Technique) works by randomly picking a point from the minority class and computing the k-nearest neighbors for this point. The synthetic points are added between the chosen point and its neighbors.

In [72]:
## Resampling:
# import library
from imblearn.over_sampling import SMOTE

smote = SMOTE()

X = training_df.drop('compliance', axis=1)
y = training_df.compliance
# fit predictor and target variable
X_smote, y_smote = smote.fit_resample(X, y)
In [73]:
y_smote.value_counts()
Out[73]:
0.0    148282
1.0    148282
Name: compliance, dtype: int64

the minority class is resampled, now the class imbalance is addressed and there are equal number of elements in both the classes

In [74]:
# Test train split
# default is 75% / 25% train-test split
X_train, X_test, y_train, y_test = train_test_split(X_smote, y_smote, random_state=0)

Model Building ¶

Logistic Regression¶

In [75]:
from sklearn.linear_model import LogisticRegression

log_reg_smote = LogisticRegression(max_iter=10000).fit(X_train, y_train)

#get predictions
y_pred = log_reg_smote.predict(X_test)

# predict probabilities
y_probs = log_reg_smote.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [76]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);
In [77]:
print("confusion matrix")
pd.crosstab(y_test, y_pred, rownames = ['Actual'], colnames =['Predicted'])

confusion matrix
Out[77]:
Predicted 0.0 1.0
Actual
0.0 33142 4089
1.0 15263 21647
In [78]:
import matplotlib.pyplot as plt  
from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(log_reg_smote, X_test, y_test)
plt.show()

Gradient-boosted decision trees¶

In [79]:
from sklearn.ensemble import GradientBoostingClassifier

GBC_D3_smote = GradientBoostingClassifier(random_state = 0)
GBC_D3_smote.fit(X_train, y_train)

#get predictions
y_pred = GBC_D3_smote.predict(X_test)

# predict probabilities
y_probs = GBC_D3_smote.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [80]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Decision tree classifier¶

In [81]:
from sklearn.tree import DecisionTreeClassifier

dtc_smote = DecisionTreeClassifier(max_depth=3).fit(X_train, y_train)

#get predictions
y_pred = dtc_smote.predict(X_test)

# predict probabilities
y_probs = dtc_smote.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [82]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Random Forests¶

In [83]:
from sklearn.ensemble import RandomForestClassifier

rf_smote = RandomForestClassifier(min_samples_split=100, max_depth=5, max_features = 8, random_state = 0).fit(X_train, y_train)

#get predictions
y_pred = rf_smote.predict(X_test)

# predict probabilities
y_probs = rf_smote.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [84]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

XG Boost Classifier¶

In [85]:
from xgboost import XGBClassifier

xgb_smote = XGBClassifier(eval_metric='logloss').fit(X_train, y_train)

#get predictions
y_pred = xgb_smote.predict(X_test)

# predict probabilities
y_probs = xgb_smote.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [86]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Logistic regression: Data Normalization¶

In [87]:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train_scaled = scaler.fit_transform(X_smote)
X_train_norm, X_test_norm, y_train_norm, y_test_norm = train_test_split(X_train_scaled, y_smote, random_state=0)

log_norm_reg_smote = LogisticRegression(max_iter=10000).fit(X_train_norm, y_train_norm)

#get predictions
y_pred = log_norm_reg_smote.predict(X_test_norm)

# predict probabilities
y_probs = log_norm_reg_smote.predict_proba(X_test_norm)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test_norm, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [88]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Random under-sampling with imblearn¶

RandomUnderSampler is a fast and easy way to balance the data by randomly selecting a subset of data for the targeted classes. Under-sample the majority class(es) by randomly picking samples with or without replacement.

In [89]:
import imblearn
from imblearn.under_sampling import RandomUnderSampler
In [90]:
X = training_df.drop('compliance', axis=1)
y = training_df.compliance

# fit predictor and target variable
rus = RandomUnderSampler(random_state=0, replacement=True)# fit predictor and target variable
X_rus, y_rus = rus.fit_resample(X, y)
In [91]:
X_train, X_test, y_train, y_test = train_test_split(X_rus, y_rus, random_state=0)
In [92]:
y_rus.value_counts()
Out[92]:
0.0    11597
1.0    11597
Name: compliance, dtype: int64

the majority class is resampled, now the class imbalance is addressed and there are equal number of elements in both the classes

Logistic regression¶

In [93]:
from sklearn.linear_model import LogisticRegression
# default is 75% / 25% train-test split

log_reg_rus = LogisticRegression(max_iter=10000).fit(X_train, y_train)

#get predictions
y_pred = log_reg_rus.predict(X_test)

# predict probabilities
y_probs = log_reg_rus.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [94]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Gradient-boosted decision trees¶

In [95]:
from sklearn.ensemble import GradientBoostingClassifier

GBC_D3_rus = GradientBoostingClassifier(random_state = 0)
GBC_D3_rus.fit(X_train, y_train)

#get predictions
y_pred = GBC_D3_rus.predict(X_test)

# predict probabilities
y_probs = GBC_D3_rus.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [96]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Decision tree classifier¶

In [97]:
from sklearn.tree import DecisionTreeClassifier

dtc_rus = DecisionTreeClassifier(max_depth=3).fit(X_train, y_train)

#get predictions
y_pred = dtc_rus.predict(X_test)

# predict probabilities
y_probs = dtc_rus.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [98]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Random Forests¶

In [99]:
from sklearn.ensemble import RandomForestClassifier

rf_rus = RandomForestClassifier(min_samples_split=100, max_depth=5, max_features = 8, random_state = 0).fit(X_train, y_train)

#get predictions
y_pred = rf_rus.predict(X_test)

# predict probabilities
y_probs = rf_rus.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [100]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

XG Boost Classifier¶

In [101]:
from xgboost import XGBClassifier

xgb_rus = XGBClassifier(eval_metric='logloss').fit(X_train, y_train)

#get predictions
y_pred = xgb_rus.predict(X_test)

# predict probabilities
y_probs = xgb_rus.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [102]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Logistic regression: Data Normalization¶

In [103]:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train_scaled = scaler.fit_transform(X_rus)
X_train_norm, X_test_norm, y_train_norm, y_test_norm = train_test_split(X_train_scaled, y_rus, random_state=0)

log_norm_reg_rus = LogisticRegression(max_iter=10000).fit(X_train_norm, y_train_norm)

#get predictions
y_pred = log_norm_reg_rus.predict(X_test_norm)

# predict probabilities
y_probs = log_norm_reg_rus.predict_proba(X_test_norm)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test_norm, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [104]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Random Forest Classifier with ‘class_weight’ parameter ¶

Random Forrest Classifier has a parameter “class_weight” to specify the weights of each class in case of an imbalanced dataset. In the sample dataset, class 0 is approx 13 times more prevalent than class 1. Hence, we will mention the “class weights” in such proportion for the algorithm to compensate during training.

In [105]:
X = training_df.drop('compliance', axis=1)
y = training_df.compliance

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
In [106]:
from sklearn.ensemble import RandomForestClassifier

weighted_rf = RandomForestClassifier(min_samples_split=100, max_depth=5, random_state=0,class_weight={0:13, 1:1}).fit(X_train, y_train)

#get predictions
y_pred = weighted_rf.predict(X_test)

# predict probabilities
y_probs = weighted_rf.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [107]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

BalancedRandomForestClassifier in imbalanced-learn library ¶

A balanced random forest randomly under-samples each boostrap sample to balance it.

In [108]:
from imblearn.ensemble import BalancedRandomForestClassifier

brfc = BalancedRandomForestClassifier(min_samples_split=100, max_depth=5,n_estimators=200,random_state=0).fit(X_train,y_train)

#get predictions
y_pred = weighted_rf.predict(X_test)

# predict probabilities
y_probs = weighted_rf.predict_proba(X_test)

# keep probabilities for the positive outcome only
y_probs = y_probs[:, 1]

# calculate roc curve
fpr_, tpr_, _ = roc_curve(y_test, y_probs)

# calculate AUC
auc_ = roc_auc_score(y_test, y_probs)

#confusion matrix
cm=confusion_matrix(y_test,y_pred)

specificity_ = cm[0,0]/(cm[0,0]+cm[0,1]) #TN/(TN+FP)
In [109]:
#append to df scores
add_row(scores, [accuracy_score(y_test, y_pred), recall_score(y_test, y_pred), precision_score(y_test, y_pred),
                 f1_score(y_test, y_pred), specificity_, auc_, tpr_, fpr_]);

Model performance paramter comparison¶

In [110]:
# Add the model names as column 
Model = ['Dummy Classifier', 'Logiestic Regression-SMOTE', 'Gradient Boost D3-SMOTE', 'Decision Tree-SMOTE',
         'Random Forest-SMOTE', 'XG Boost-SMOTE', 'Logistic Regression Normalised-SMOTE', 'Logiestic Regression-RUS',
         'Gradient Boost D3-RUS', 'Decision Tree-RUS', 'Random Forest-RUS', 'XG Boost-RUS',
         'Logistic Regression Normalised-RUS', 'Random Forest-CW', 'Random Forest-BRFC']
  
scores.insert(0, "Model", Model, True)
In [111]:
scores.sort_values(by='AUC', ascending=False, inplace=True)
In [112]:
scores.reset_index(drop=True, inplace=True)
In [113]:
scores
Out[113]:
Model Accuracy Recall score Precision score F1 score Specificity AUC TPR FPR
0 XG Boost-SMOTE 0.761697 0.661338 0.825276 0.734268 0.861191 0.838339 [0.0, 2.70929287455974e-05, 0.0002709292874559... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
1 Gradient Boost D3-SMOTE 0.745269 0.634381 0.812851 0.712612 0.855201 0.813505 [0.0, 2.70929287455974e-05, 5.41858574911948e-... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
2 Random Forest-SMOTE 0.741466 0.598998 0.835058 0.697599 0.882705 0.807333 [0.0, 5.41858574911948e-05, 8.12787862367922e-... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
3 XG Boost-RUS 0.735472 0.620678 0.798652 0.698506 0.847411 0.798779 [0.0, 0.0017464198393293748, 0.002444987775061... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
4 Logiestic Regression-SMOTE 0.738984 0.586481 0.841118 0.691090 0.890172 0.797036 [0.0, 0.002492549444594961, 0.0025467353020861... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
5 Logistic Regression Normalised-SMOTE 0.738107 0.579220 0.846190 0.687704 0.895625 0.796266 [0.0, 0.000135464643727987, 0.0003251151449471... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
6 Gradient Boost D3-RUS 0.729609 0.576668 0.822621 0.678029 0.878747 0.795219 [0.0, 0.001047851903597625, 0.0013971358714634... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
7 Random Forest-RUS 0.730816 0.575271 0.826807 0.678476 0.882493 0.793759 [0.0, 0.010478519035976248, 0.0220048899755501... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
8 Decision Tree-SMOTE 0.734182 0.570306 0.845449 0.681142 0.896645 0.787136 [0.0, 0.06261175833107559, 0.09336223245732864... [0.0, 0.00026859337648733585, 0.00042974940237...
9 Random Forest-CW 0.931449 0.057527 0.976608 0.108653 0.999892 0.785195 [0.0, 0.0003444712366517396, 0.001722356183258... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
10 Random Forest-BRFC 0.931449 0.057527 0.976608 0.108653 0.999892 0.785195 [0.0, 0.0003444712366517396, 0.001722356183258... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
11 Logiestic Regression-RUS 0.727367 0.555362 0.837724 0.667927 0.895095 0.780769 [0.0, 0.00209570380719525, 0.01047851903597624... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
12 Logistic Regression Normalised-RUS 0.724263 0.550472 0.834746 0.663439 0.893733 0.778093 [0.0, 0.00034928396786587494, 0.00488997555012... [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
13 Decision Tree-RUS 0.725470 0.553615 0.834650 0.665687 0.893052 0.767860 [0.0, 0.05902899056933287, 0.09640237513098149... [0.0, 0.0, 0.0003405994550408719, 0.0895776566...
14 Dummy Classifier 0.927371 0.000000 0.000000 0.000000 1.000000 0.500000 [0.0, 1.0] [0.0, 1.0]

plot model performance parameters ¶

In [114]:
cols=['AUC', 'Accuracy', 'Recall score', 'Precision score', 'Specificity', 'F1 score']
plot_data=scores[cols]
In [115]:
# Create just a figure and only one subplot
plt.figure(figsize = (16,9))

sns.lineplot(data=plot_data,dashes = False)
plt.title("Model performance parameters", fontsize = 20) # for title
# Adding label on the y-axis
plt.ylabel('Numbers label')

plt.xticks(np.arange(0, (len(plot_data)+1), 1))

plt.show()
In [116]:
#scores.to_csv('model_scores.csv')

Based on the model performance scores, we can understand that XG Boost with Synthetic Minority Over-Sampling Technique has highest AUC of 0.85. Also the model has an accuracy rate of 0.77. This is also one of the models with high recall of 0.68

We can use this model on our training dataset

get probability compliance for Test Data¶

In [117]:
probability=xgb_smote.predict_proba(testing_df)[:,1]
# we need to select the probability that the ticket will be paid
In [118]:
final_result=pd.DataFrame({'ticket_id':test_ticketid['ticket_id'], 'probability':probability})
In [119]:
final_series=pd.Series(probability, index = test_ticketid['ticket_id'])

!reference 1..
!reference 2..
!reference 3..
!reference 4..

In [ ]:
In [ ]: